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Overview 


Improving the quality of early care and education (ECE) is a central goal of state and national initiatives 
aimed at supporting children’s early learning and healthy development. Given the prevalence of these 
investments, program administrators, policymakers, and researchers are interested in understanding how 
quality improvement activities can best target changes in practice that will support children’s development, 
especially those that can be sustained over time across a wide range of programs. Advances in the science 
of improvement offer the ECE field a new framework for thinking about approaches to quality improvement. 
While some specific methods derived from improvement science have generated dialogue and investments 
in the ECE field, these methods are not yet widespread in practice. 


The Breakthrough Series Collaborative (BSC) is a structured improvement methodology developed 
originally for use in the health care field. The goal of the BSC is to build organizational capacity for quality 
improvement; support data-informed decision making; engage leadership from all levels of an organization 
in the change process; and ultimately improve quality, outcomes, and efficiency (Mittman, 2004). The 
Culture of Continuous Learning (CCL) Project: A Breakthrough Series Collaborative for Improving Child Care 
& Head Start Quality will examine the potential of the BSC to support the organizational capacity of child 
care and Head Start settings to test and implement program improvements. 


Purpose 


This literature review will examine the evidence base of the BSC to better understand its outputs and 
outcomes, as well as the contextual factors that may support success. The review’s findings will be 
incorporated into the next phase of the CCL Project, which includes a feasibility study to examine the 
implementation of the BSC in both Head Start and child care settings. The results of this study will help 
determine whether the BSC holds promise as a structured approach to improvement in ECE settings, along 
with the feasibility of implementing the BSC in different types of ECE programs. Results will also be used to 
inform planning for future research related to quality improvement in ECE. 


Methods 


Literature included in this review was identified through Google Scholar and EBSCOhost, including the 
following databases: Academic Search Complete, Consumer Health Complete, Google Scholar, Education 
Research Complete, ERIC, Health Source, PsycINFO, and Psychology and Behavioral Sciences Collection. 
The following search terms were used: Continuous Quality Improvement (CQI), Breakthrough Series 
Collaborative (BSC), Plan-Do-Study-Act Cycles (PDSA), Learning Collaboratives, Improvement Science, and 
Implementation Science. 


Literature included peer-reviewed research articles, policy briefs, research reports, and white papers. The 
literature cited in this review was published between 1998 and 2016, although the team prioritized research 
conducted within the last 10 years. Sections | through V of this report include a mix of literature related 

to the BSC and other quality improvement methods relevant for early childhood, while Section V includes 
literature specific to the outcomes related to BSC implementation. The literature search specific to the 
BSC identified 49 articles, of which 34 included a focus on BSC outcomes. These articles largely focused 
on studies conducted in the health care field. Four members of the Child Trends team analyzed the articles 
using a Summary table to compile findings related to BSC outcomes. 
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Key Findings and Highlights 


Overall, the literature examining the implementation and outcomes indicates that the BSC methodology has 
consistent evidence of improving practices in healthcare, child welfare, and mental health service provision 
by facilitating the implementation of evidence-based practices (EBP) and building the infrastructure to 
support quality improvement efforts. Although outcomes at the recipient level were either not reported or 
mixed, the overall findings from this review indicate that the BSC model holds promise for improving the 
quality of organizational processes and individual practice in the ECE sector. 


The literature suggests that the CCL Project may need to: 


¢ Support buy-in and engagement in the BSC at multiple levels of the system 

¢ Ensure supports for implementation, including training, supervision, and data systems 

¢ Develop strategies for building capacity to engage in the improvement process at all levels 
e¢ Examine ongoing incentives for supporting participation in the initiative 

¢ Identify strategies for trouble-shooting common problems that arise 
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Culture of Continuous Learning Project: 
A Literature Review of the Breakthrough 
Series Collaborative (BSC) 


Introduction 


Improving the quality of early care and education (ECE) is a central goal of state and national initiatives 
aimed at supporting children’s early learning and healthy development. For example, in federal fiscal year 
2014, states spent $958 million on quality improvement activities through the Child Care and Development 
Fund (CCDF).' In the same year, Head Start spent $229 million on training, technical assistance, and 
program support.” Given these investments, program administrators, policymakers, and researchers are 
interested in understanding how quality improvement activities can best target changes in practice that 

will impact children’s development, especially those that can be sustained over time across a wide range of 
programs (Tout, Epstein, Soli, & Lowe, 2015). The ECE field is eager for innovative strategies that accelerate 
and enhance existing quality improvement efforts. Of particular interest are approaches that address the 
need for capacity-building in ECE organizations, support the testing and implementation of evidence-based 
practices (EBPs) among ECE practitioners, and focus on developing a culture of continuous learning (Tout et 
al., 2015; Derrick-Mills, 2015, Wiggins, & Mathias, n.d.). 


Advances in the science of improvement offer the ECE field a new framework for thinking about approaches 
to quality improvement. For the purposes of this paper, improvement science is defined as the study of 
methods, tools, and content that produce improvements in practice (Bryk, Gomez, Grunow, & LeMahieu, 
2015). Improvement science examines theories and methods for developing, testing, and implementing 
changes that result in improvement (Langley, Moen, Nolan, Nolan, Norman, & Provost, 2009). It incorporates 
insights from implementation science, systems theory, behavioral science, knowledge development, and 


‘For more information see: http://www.acf.hhs.gov/occ/resource/ccdf-expenditures-overview-for-fy-2014-all-appropriation-years 
?For more information see: https://eclkc.ohs.acf.hhs.gov/hslc/data/factsheets/docs/hs-program-fact-sheet-2014.pdf 
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change management. Improvement methods have been 
used widely in sectors such as health care to facilitate 
ongoing quality improvements that are tailored to local 
conditions, and use data and a process of inquiry to 
analyze and improve practice (Bailie, Si, O Donoghue, 

& Dowden, 2007; O'Neill et al., 2011). A key lesson 

from improvement science is that the ability to make 
improvements is driven by the combination of content 
knowledge and knowledge about how to improve. While 
some specific methods derived from improvement 
science have generated dialogue and investments in 
pockets of the ECE field, these methods are not yet 
widespread in practice. 


The Culture of Continuous Learning (CCL) Project is supported by the federal Office of Planning, Research 
and Evaluation in the Administration for Children and Families, and tests the feasibility of implementing 

an improvement science methodology in ECE settings. Child Trends partnered with the University of 
Massachusetts, Boston (UMB), JRA Consulting Ltd., and the Center for the Study of Social Policy to conduct 
the CCL Project, selecting the Breakthrough Series Collaborative (BSC) as the specific quality improvement 
method to test (Institute for Healthcare Improvement [IHI], 2003). 


The BSC methodology is a structured improvement methodology developed originally for use in the health care 
field. The goal of the BSC is to build organizational capacity for quality improvement; support data-informed 
decision making; engage leadership from all levels of an organization in the change process, and ultimately, 
improve quality, outcomes, and efficiency (Mittman, 2004). BSCs have led to documented positive results 

in health care, and have been tested in social service fields such as child welfare to engage organizations in 
collaborative learning and changes in practice that lead to measurable improvement (IHI, 2003). 


The CCL Project will examine the BSC’s potential to support the organizational capacity of child care and 
Head Start settings to test and implement program improvements. First, the project will implement a year- 
long BSC with a focus on practices that promote children’s social and emotional learning, and will study the 
feasibility of implementing the BSC in eight ECE centers. Second, the CCL Project will produce a rigorous 
evaluation plan to examine: (1) the effectiveness of a BSC in supporting and sustaining evidence-based 
practices in social and emotional learning (or other developmental domains); (2) sustaining an organizational 
culture of quality improvement; and (3) the effect of these improvements on children’s development. Finally, 
the CCL Project will inform recommendations about how the BSC could be implemented in more programs 
across diverse geographic areas—perhaps with additional program types such as family child care and/or 
state pre-kindergarten programs. 


This literature review will summarize the empirical foundations for the BSC, and the outputs and outcomes 
of the BSC process documented in the literature. This paper first provides a grounding in improvement 
science literature, which can be used to guide the CCL Project in assessing the feasibility of BSC 
implementation in ECE settings. The review also provides background on quality improvement approaches 
used in ECE settings, along with an overview of the BSC model. The second half of the report provides an 
overview of the outputs and outcomes identified from prior studies of the BSC. The report closes with a 
brief synthesis of considerations for next steps for the CCL Project. This report is organized as follows: 


|. The Need for New Approaches to Quality Improvement in Early Care and Education Settings 
Il. Overview of Improvement Science 
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Ill. Overview of the Breakthrough Series Collaborative 

IV. A Review of the Literature: BSC Outputs and Outcomes 
V. Considerations for the CCL Project Next Steps 

VI. Conclusion 


Methods 


Literature included in this review was identified through Google Scholar and EBSCOhost, including the 
following databases: Academic Search Complete, Consumer Health Complete, Google Scholar, Education 
Research Complete, ERIC, Health Source, PsycINFO, and Psychology and Behavioral Sciences Collection. 
The following search terms were used: Continuous Quality Improvement (CQI), Breakthrough Series 
Collaborative (BSC), Plan-Do-Study-Act Cycles (PDSA), Learning Collaboratives, Improvement Science, 

and Implementation Science. Literature included peer-reviewed research articles, policy briefs, research 
reports, and white papers. The literature cited in this review was published between 1998 and 2016, although 
research conducted within the last 10 years was prioritized. 


Sections | through V of this report include a mix of literature from peer-reviewed journals (using the search 
terms and databases noted), as well as reports, white papers, and policy briefs related to the BSC and other 
quality improvement methods relevant for the early childhood field. Section V includes literature specific to 
the outcomes related to BSC implementation. The literature search specific to the BSC identified 49 articles, 
of which 34 included a focus on BSC outcomes. These articles largely focused on studies conducted in the 
health care field. Four members of the Child Trends team analyzed the articles using a summary table to 
compile findings related to BSC outcomes. 


I. The Need for New Approaches to Quality Improvement in 
Early Care and Education Settings 


Describing quality improvement approaches currently used in ECE settings provides important context for 
the CCL Project. This section includes a brief overview of current quality improvement focal areas, along 
with approaches typically used in child care and Head Start settings, and discusses the unique realities and 
challenges ECE programs have encountered in their quality improvement efforts. These challenges provide 
insight into the rationale for exploring new and innovative approaches for supporting the process of quality 
improvement in early care and education settings. 


Current Areas of Focus and Approaches to Quality Improvement in ECE 


Broadly speaking, quality improvement in ECE settings fall into one of three areas: (1) system- or state-level; 
(2) program-level; and (3) practitioner- or staff-level (Boller, Tarrant, & Schaack, 2014). 


Systems-level quality improvement approaches include efforts to define quality standards for ECE and to 
promote structures that support movement toward higher quality. Systems-level strategies include state 
Quality Rating and Improvement Systems (QRIS) that set tiered quality indicators, measure and rate the 
quality of ECE programs, and provide coaching and other incentives to help programs achieve higher quality. 
Other quality improvement areas of focus at the system or state level include professional development 
supports (such as the development of certification and credential programs to improve workforce 
competencies), governance structures to improve coordination of supports for ECE, and data systems that 
can be used to monitor progress toward goals for the ECE system. 
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Head Start has a long history of investing in system-level quality improvement efforts, including significant 
ongoing investments in training and technical assistance and, more recently, program performance 
standards around alignment with state licensing and QRIS. Similarly, the recently reauthorized Child Care 
and Development Block Grant (CCDBG) of 2014 includes an increase to the minimum quality set-aside 
(from 4% to 9% with an additional 3% spent on provisions for infants and toddlers). Grantees have great 
flexibility in how they spend these CCDBG quality dollars to strengthen local systems. For example, their 
quality improvement (QI) approaches for this area of focus might include funding activities to support a 
state QRIS, focusing on child care licensing efforts, or providing technical assistance (such as coaching and 
consultation) and training, among other areas. 


Program-level quality improvement strategies are typically focused on improving structural and process 
quality in order to promote positive outcomes for children. Structural quality includes features of the 
physical environment such as health and safety provisions, the instructional materials available to children, 
and child-to-adult ratios and group size. This aspect of quality can also include a focus on improving center- 
and home-based providers’ business practices. Process quality includes aspects such as the quality of 

the interactions between and among adults and children. Both structural and process quality are typically 
included in the program-level quality addressed through participation in a QRIS, with programs being rated 
on a variety of quality indicators (including administration and management, the learning environment, 
family engagement, curriculum, and assessment practices). Attaining national accreditation is another 
program-level strategy for quality improvement. Head Start programs are required to meet program-level 
standards with regular monitoring visits from the Office of Head Start. 


Quality improvement strategies designed to support ECE practitioners/staff directly target the skills and 
competencies of individuals. Typical methods include improving care and instructional practices through 
content-specific trainings/workshops, onsite coaching, and supporting professionals’ attainment of formal 
education. In most systems, program- and practitioner-level quality improvement strategies are conducted 
in a largely parallel fashion. Head Start training and technical assistance addresses both program-level and 
practitioner-/staff-level elements of quality, but these efforts need not be explicitly linked. Similarly, in QRIS, 
coaching tends to focus on either the director or the teachers in an ECE program rather than an intentional 
focus on the organization as a whole. For example, with limited time and resources for on-site support, 
consultants in one quality improvement initiative focused heavily on engaging directors in developing a 
quality improvement plan. While teachers also participated in the initiative, they reported lower levels of 
satisfaction with their experience (Warner-Richter et al., 2016). 


Growing Interest in New Approaches to Quality Improvement 


While current approaches to quality improvement have strengths and successes (e.g., Warner-Richter et 
al., 2016), the state of the field also suggests a growing interest in new and evidence-based approaches 

to quality improvement. The last decade has seen shifts toward QI approaches that acknowledge the 
continuous or ongoing nature of quality improvement in ECE settings, sometime referred to as Continuous 
Quality Improvement (CQI; Wiggins & Mathias, n.d.). CQIl is defined as an ongoing and cyclical process in 
which practitioners use data to identify strengths and opportunities for improvement, then test, implement, 
learn from, and revise solutions (National Child Welfare Resource Center for Organizational Improvement 
and Casey Family Programs, 2005). The term CQI, however, is often used in different ways in ECE. The way 
CQI is implemented and supported varies significantly across ECE systems. In addition, while the use of CQI 
terminology and strategies has grown in ECE settings in recent years, these efforts do not yet utilize the 
comprehensive, capacity-building, and systematic approaches characterized by improvement science. This 
section of the report describes the shift in quality improvement approaches that utilize elements of a CQI 
frame, such as using to data inform goal setting, then describes the gaps in these practices in relation to the 
principles of improvement science. These principles are described in more detail in the next section. 
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CQI in child care 


The term CQI was introduced to child care primarily in the context of state Quality Rating and Improvement 
Systems (QRIS). QRIS often require the development of a quality improvement plan that structures a 
program’s priorities and helps to establish a process for monitoring progress and reassessing goals over 
time. The growing interest in supporting work on quality improvement plans in QRIS motivated a national 
technical assistance response to help support state-level implementation. In 2014, the BUILD Initiative 
hosted a professional development opportunity that included an in-person meeting for state QRIS 
administrators, interactive web-based sessions, and access to an electronic collection of resources related 
to CQI. In 2016, the BUILD initiative also began hosting a CQI Community of Practice. 


Despite increased awareness and support related to CQI, the term is used in state QRIS to refer to a 
relatively narrow set of practices. For example, a review of how CQI is addressed in QRIS quality indicators 
reveals that CQI is operationalized typically as programs’ use of self-assessments and the development of 
quality improvement plans based on assessment results.? In at least two states, however, a CQI approach 
is being implemented in a more systematic way. In New Mexico, the QRIS system and consultant model 
are designed to support ECE programs (including child care centers, family child care programs, Head 
Start programs, and New Mexico Pre-K programs) in engaging in regular cycles of continuous quality 
improvement (going beyond development of a point-in-time plan). Oregon is another state that has 
embraced a CQI approach and focuses on improvement activities at both the system and program levels 
(through the use of data collection and feedback loops).* Both of these efforts are still in the early stages 
of implementation, so insights on the effectiveness of these approaches and the adequacy of supports to 
implement CQI through a QRIS are not yet known. A process evaluation of the Oregon CQI approach is 
currently underway, as is an evaluation of the New Mexico FOCUS QRIS and effectiveness of CQI supports. 


CQI in Head Start 


Head Start program policies have also shifted to promote and incentivize CQI practices. For example, as 
aresult of the Improving Head Start for School Readiness Reauthorization Act of 2007, Head Start has 
increasingly emphasized the use of data for program improvement (Derrick-Mills, Sandstrom, Pettijohn, 
Fyffe, & Koulish, 2014). Head Start programs must also conduct annual self-assessments to monitor 
progress toward program goals. The Advisory Committee on Head Start Research and Evaluation (U.S. 
Department of Health and Human Services, 2012) recommended that Head Start become a “learning 
organization where decisions about instructional practices and curricula, assessments, monitoring, 
professional development, and organizational development are integrally linked to each other and to 
school readiness and other key goals” (p. 8). The Head Start Program Performance Standards, revised in 
2016, reflect one way in which Head Start is moving toward its goal of becoming a “learning organization”: 
the standards place an increased and explicit focus on using data for quality improvement in Head Start 
programs. 


Evidence from recent studies of Head Start (e.g., Derrick-Mills, 2015) suggests that Head Start programs 
are using data for program improvement, although there is still progress to be made. The Head Start 
Leadership, Excellence, and Data Systems (LEADS) project (Derrick-Mills, 2015) reviewed existing literature 
in other relevant fields to identify the organizational factors that foster effective use of data for program 
improvement. Drawing from this body of literature and the knowledge of an expert panel group, the 

LEADS project developed a conceptual framework highlighting eight key elements conducive for fostering 
an organization’s use of data for continuous quality improvement, including: leadership, commitment to 


3 Based on Child Trends’ analysis of the 2016 QRIS Compendium. 
4 For more information see: http://qrisnetwork.org/member/calendar/event/160127/a-multi-level-cqi-approach-using-qris-participation-data- 
inform-system-. 
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resources, analytic capacity, professional development, culture of inquiry, continuous cycle, organizational 
context, and external environment (Figure 1). This framework highlights the importance of many internal 
organizational characteristics that are key to continuous quality improvement (e.g., transformational 
leadership, safe space for a culture of inquiry), while also acknowledging the influences of the surrounding 
environment (e.g., government mandates, existing professional development systems). 


Although the study did not assess these organizational and environmental factors within Head Start 
programs, the authors identified some of the challenges in implementing quality improvement strategies 

in early education settings (Derrick-Mills, 2015). For instance, analytic capacity has been identified as an 
important facilitator for the effective integration of CQI approaches, but individual ECE program leaders 
may not necessarily see it as their responsibility—or have sufficient access to professional development 
resources to learn how to collect, analyze, and use data in their practices. As a result, early education 
settings like Head Start may face challenges in setting up systems to collect and use data in ways that are 
efficient and useful to programs, both for shorter- and longer-term goals. This framework also calls for a 
shift from using data primarily for compliance with regulations, to using data as part of a culture of learning 
and continuous improvement. The Head Start Designation Renewal System (DRS) is an additional context 
for consideration of CQI in Head Start programs. The DRS requires grantees to re-compete for their funding 
if they do not meet a set of quality standards, including threshold scores on the Classroom Assessment and 
Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008); as a result, Head Start grantees have regular 
access to data from the CLASS that may be used to inform quality improvements (Derrick-Mills et al., 2016). 


Figure 1. LEADS Continuous Quality Improvement Conceptual Framework 
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Gaps in Current Approaches to Quality Improvement 


Evidence from evaluations of state QRIS indicates some early successes in engaging ECE programs in 
overall quality improvement activities and supporting advancement in QRIS quality levels (e.g., Karoly et al., 
2016; Cleveland et al, 2016). However, closer examination of program quality data reveals that programs may 
not be improving on the instructional dimensions of quality that are most likely to support young children’s 
development (e.g., Tout et al, 2016). For example, a recent study of Head Start classroom quality assessed 

in 2006, 2009, and 2014 found that scores on both the Early Childhood Environment Rating Scale — Revised 
(ECERS-R) and domain scores on the CLASS increased over time (Aikens, Bush, Gleason, Malone, & Tarullo, 
2016). However, nearly 50 percent of classrooms received scores in the minimal range on the ECERS-R 
indicators related to learning activities and teacher-child interactions (Aikens et al., 2016). Likewise, national 
Head Start grantee data indicates that most classrooms scored in the low range on CLASS Instructional 
Support dimensions (Office of Head Start, 2015). Similarly, in the diverse sample of early childhood settings 
in the National Institute of Child Health and Human Development (NICHD) Study of Early Child Care and 
Youth Development, most provide care that is “fair” (between “poor” and “good”)—particularly in regard to 
process features of quality considered to have the most proximal influence on child development (NICHD 
Early Child Care Research Network, 2005). 


As noted, many current approaches that have been utilized by ECE programs focus on individual 
improvement rather than programmatic or system-level improvement. As such, these approaches may be 
missing opportunities to focus on the features of organizational culture that can promote and sustain quality 
improvement—not only the level of the staff/practitioner, but also the program and systems levels. This gap 
sets the stage for new approaches that can maximize sustainable impact from the resources invested. 


Another set of challenges is related to the capacity of ECE programs to engage in sustained and meaningful 
quality improvement processes, given staff or leadership turnover, low wages, lack of planning time outside 
of their time with children, and low levels of education and training on ECE pedagogy. In one recent 
evaluation of a quality improvement initiative for child care centers in Philadelphia, center participants that 
enrolled in the initiative had a variety of documented staffing and infrastructure challenges that limited their 
capacity to fully participate in the initiative (Warner-Richter et al., 2016). Even when there is agreement that 
quality improvement is a worthy goal, technical assistance agencies and center staff may underestimate the 
scope of engagement and participation needed to achieve real and lasting change. The realities of current 
QI efforts in ECE settings provide an important context for exploring new approaches and methods for 
improvement in Head Start and child care settings. 


Interest in approaches to CQI in ECE is growing, but, in practice, CQI still means different things to 
different audiences. As a result, implementation may vary significantly across settings. Early care and 
education program leaders may develop a quality improvement plan but the extent to which this plan 
guides organizational learning and results in measurable quality improvement has not been established. A 
recent report sponsored by the Carnegie Foundation stated a similar conclusion when reviewing activities 
characterized as “continuous improvement” in education, finding that improvement planning (e.g., strategic 
planning) too often ignores issues related to process, measurement, and contextualization of outcomes 
(Park, Hironaka, Carver, & Nordstrum, 2013). In contrast, in improvement science, “continuous quality 
improvement” typically refers to the structured use of inquiry cycles by collaborative teams to build 
knowledge for improvement in the context of organizations that are curious and open to learning. These 
inquiry cycles—sometimes called PDSA cycles—involve groups of professionals (and sometimes parents) in 
testing new practices and collecting data to examine whether the tests result in the desired improvement. 
This process then informs ongoing refinement and testing of strategies to achieve the desired outcomes, 
spreading those strategies throughout a program or center and sustaining them (“holding the gains”) over 
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time through organizational structures, culture, and leadership. These features of the improvement science 
approach to CQI, coupled with the need to address organizational culture and structures, hold promise as a 
potential approach for use in ECE settings, and are thus the focus of the CCL Project. 


Il. An Overview of Improvement Science 


CQI is a term that has been used in ECE to describe a wide spectrum of activities and models, from 
practitioner inquiry cycles to data-driven decision making to the use of a written annual self-assessment 
and improvement plan. This terminology challenge can lead to confusion, and interferes with the ability 

to specify and determine what kinds of quality improvement methods are effective. In general, ECE 
researchers, practitioners, and quality improvement professionals have traditionally focused exclusively on 
what is to be changed, such as teaching practices in areas like language and literacy or social skills, without 
equal attention on how to make change in ways that lead to sustained improvement (although changes in 
this paradigm are beginning to take hold). As noted, the Head Start LEADS initiative has highlighted the 
importance of using data for CQI in early care and education settings. Similarly, one of the core features of 
the Educare model is data utilization to inform continuous quality improvement. Improvement science offers 
rich insights and research evidence about CQI methods for improving practice more specifically; it offers 
theoretical grounding and practical tools that can deepen an understanding of quality improvement and 
potentially help ECE practitioners “get better at getting better” (Bryk, Gomez, Grunow, & LeMahieu, 2015). 


Improvement science examines theories and methods for developing, testing, and implementing 
organizational or systems-wide changes that result in improvement (Langley, Moen, Nolan, Nolan, 

Norman, & Provost, 2009). Improvement science incorporates insights from multiple theories, including 
implementation science, systems theory, behavioral science, knowledge development, and change 
management. Ideas about continuous quality improvement grew in the business sector in the 1930s, and 
have evolved and spread over time into healthcare and other fields as a strategy to ensure high-quality 
practices (Bailie et al., 2007; Brown & Marshall, 2008). The improvement science approach aims to facilitate 
ongoing quality improvement in a way that is tailored to local conditions, by using objective data to analyze 
and improve practice (Bailie et al., 2007; O’Neill et al., 2011). This process is dependent on the ongoing 
collection of data to inform goals and strategies for improvement. It involves developing and implementing 
one or more improvement strategies and regularly examining data on system or personnel practices or 
clients to monitor what is or is not working well. This approach also assumes that both the positional leaders 
who have traditional decision-making authority and the individuals tasked with implementing changes in 
practices are key participants (Bailie et al., 2007; Brown & Marshall, 2008). 


The foundation of improvement science comes from a set of four core principles (Langley, Moen, Nolan, 
Nolan, Norman, & Provost, 2009). 


1. Individuals and organizations operate in the context of existing systems. If organizations are to 
achieve better results, they must address the whole system, not just a portion. Individuals such as 
teachers do not operate in isolation. Rather, they operate in the context of the workplace, in classrooms, 
and in the larger organization. To make and sustain improvement in classrooms, they may need new 
organizational structures, policies, or resources to support and enable those changes. In addition, 
organizations may be regulated and/or resourced by a larger state and federal system, and their 
capacity to change what they do is strongly influenced by the system in which they work. 


2. Strategies for improvement may incorporate a needed degree of variation. In fields like healthcare 
and education, an evidence-based strategy does not work in exactly the same way in all contexts and 
with all children. Thus, professionals must know how to implement and adapt practices so they can 
achieve the desired outcome across varied individuals and contexts. 
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3. Changes in practice require knowledge of improvement methods. The Model for Improvement, 
based on Deming’s (1986) process improvement work, helps organizations reflect on three key questions 
to guide changes in practice: 


(1) What are we trying to accomplish? (Aims) 
(2) How will we know a change is an improvement (Metrics) 
(3) What changes can we test that will result in improvement? (Strategies) 


By continuously referring to these questions and then using the basic CQI approach of PDSA cycles, 
teams rapidly test, adapt, and move strategies to full implementation. PDSAs are a structured, ongoing, 
and cyclical process, in which practitioners learn how to improve results or outcomes by testing, 
implementing, learning from, and revising solutions (National Child Welfare Resource Center for 
Organizational Improvement and Casey Family Programs, 2005). 


4. Effective working relationships are essential for improvement. Improvement science views improve- 
ment as a collaborative learning process, not a top-down effort led by a formal leader. It is inherently 
relational, fostered by respectful and reciprocal relationships and a shared goal for improvement. There- 
fore, strengthening the relationships among the different individuals and across roles in a team and an 
organization are essential for effective improvement efforts. Research on organizational learning and 
improvement emphasize the need to create a learning culture, positive relationships, high-quality com- 
munication, and psychological safety that enables individuals to speak up, be heard, and take risks when 
it comes to testing changes (Douglass, 2012; Douglass, 2014; Douglass & Klerman, 2012; Edmondson, 
1999; Gittell, 2016; Nembhard & Edmondson, 2012; Senge, 1990). 


l1l. Overview of the Breakthrough Series Collaborative 


The Breakthrough Series Collaborative (BSC) was developed by the Institute for Healthcare Improvement 
and the Associates in Process Improvement in 1995 to bridge the gap between what is known in science 
and what is actually done in practice (IHI, 2003). The effectiveness of the BSC improvement methodology 
has been established in health care settings over the last 20 years (IHI, 2003; Kaplan, Lannon, Walsh, & 
Donovan, 2010; Nadeem, Olin, Hill, Hoagwood, & Horwitz, 2013). The BSC has also been adapted by Casey 
Family Programs for use in child welfare agencies and by the National Center for Child Traumatic Stress/ 
National Child Traumatic Stress Network as a method to fully implement evidence-based treatments, with 
evidence of effectiveness (Agosti, Conradi, Halladay Goldman, & Langan, 2013). Evidence has also emerged 
from the children’s mental health field documenting that the method is promising in implementation of 
evidence-based treatments (Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2012; Haine-Schlagel, 
Brookman-Frazee, Janis, & Gordon, 2013). 


The BSC is a structured, facilitated, and collaborative methodology for CQI with foundations strongly 
rooted in improvement science. In a BSC, diverse teams from many different organizations come together 
to form a collaborative focused on achieving a specific shared goal. These teams engage in a facilitated 
mutual learning process, learn skills for improvement, and lead change within their individual organizations. 
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Models of Quality Improvement 


Improvement science has studied the development of several different models and methods for quality 
improvement that have emerged over the last several decades in the manufacturing, healthcare, human 
services, and education fields. Briefly describing these different models provides some context for 
understanding the unique goals and features of the Breakthrough Series Collaborative. 


In addition to a BSC, other well-known and widely used CQI models include LEAN, Six Sigma, Kaizan, 

and the Collaborative Improvement and Innovation Networks (CollN). LEAN was developed out of the 
automobile manufacturing industry and has been adapted by the healthcare and business sectors. LEAN is a 
quality improvement approach designed to reduce waste and errors while increasing efficiency and quality 
(Health Information Technology Research Center (HITRC); 2013; Levinson & Renick, 2002; Scoville & Little, 
2014). Like LEAN, Six Sigma and Kaizan are similar business improvement approaches that aim to reduce 
waste and errors by using data that examine and refine current practices (HITRC, 2013). ColINs are typically 
virtual teams of federal, state, and local leaders that share ideas and best practices to address a common 
problem and track their progress toward meeting collaborative goals. The Health Resources Services 
Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) has supported CollNs to address topics 
such as maternal health, newborn screening, home visiting, and pediatric emergency care. While each 
model has a slightly different purpose or focus, they all rely on similar PDSA-like processes and use data for 
continuous feedback. 


The BSC is short-term (i.e., 6 to 15 months), structured, and carefully facilitated to engage a cohort 

of program teams in focusing on improving practices, which often includes testing and implementing 
relevant evidence-based practices (EBP; for example, in the health field, reducing emergency room wait 
time, or reducing rates of emergency caesarean sections). Each team focuses on change within its own 
organization, but also works to support other teams by sharing experiences and lessons learned on 
conference calls, a group listserv, a shared website, through monthly data reporting, and at in-person 
meetings (called learning sessions). At each learning session, teams gain new skills or knowledge needed 
to accomplish the shared goal or implement the selected EBP, share experiences with cohort team 
members, and make plans for their “action period.” In between each learning session, program teams 
use PDSA cycles to make changes to current practices within their organization, assess the results, and 
spread those changes throughout the organization. The BSC process is guided by a trained facilitator 
and supported with experts in the selected content area, in addition to experts in supporting the change 
process. 


Rationale for the BSC. Several features of the BSC suggest that it may be more effective than use of 
more general CQI approaches. First, the BSC model, as used in child welfare and most recently through a 
pilot in early care and education, focuses on changing organizational cultures and beliefs by fundamentally 
changing practice. Initial organizational outcomes and practice changes were noted in the Boston Trauma- 
Informed ECE BSC (Douglass, 2017; Douglass, 2016; Boston Public Health Commission, Division of Violence 
Prevention, 2016), the Trauma-Informed Child Welfare Practice BSC (Agosti, Conradi, Halladay Goldman, 
and Langan, 2013), the California Disproportionality Project BSC (Agosti, J., 2011), the CA Independent 
Living Transformation BSC (Jarboe et al., N.D.), and the Massachusetts Adolescent Permanency BSC 
(Agosti and Morrill, 2007). Second, the BSC aims to promote the sustainability of change by focusing on 
organizational capacity that can be maintained even after the intervention has concluded (e.g., Casey 
Family Programs, 2009). In addition, the BSC approach focuses on implementation in real-world settings. 
The approach emphasizes replication and sustainability in contrast to more traditional small demonstration 
or pilot projects. Throughout a BSC, leadership capacity and CQI skills are taught, modeled, coached, and 
facilitated at all levels with implementation and improvement science as the foundation. Evidence suggests 
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that teams benefit from the shared accountability of the collaborative, and the collaborative experience 
accelerates their progress in ways that will ultimately be shared, sustained, and extended into ongoing 
improvement efforts. In sum, the BSC addresses all three levels described as important for ECE quality 
improvement: system, program, and practitioner. Further, the BSC strives to address all core competencies 
described as important for a culture of continuous improvement, as cited by the LEADS’s Resource Guide for 
Head Start Programs: Moving Beyond a Culture of Compliance to a Culture of Continuous Improvement. 


The extent to which these features of the BSC resonate with child care and Head Start programs will be 
examined in the CCL Project. The section that follows provides an overview of the mechanics of the BSC 
methodology. An analysis of the literature addressing the outcomes of the BSC are described in a later 
section of this paper. 


How Is the BSC Process Implemented? 


The BSC method brings together a set of five key elements that are theorized to support effective improve- 
ment processes: (1) the change framework; (2) multi-level inclusive teams; (3) expert faculty; (4) a shared 
learning environment; and (5) the model for improvement. As noted above, a BSC is a facilitated process for 
quality improvement that brings together teams from across multiple organizations. A BSC is organized by 
an entity that provides the leadership and expertise to convene, host, and support implementation of the 
BSC process. For example, a BSC can be led by an organization, a professional association or network, a 
nonprofit entity, or a public agency. This organizing body then oversees implementation of the model and its 
five key elements described here. The staff for a BSC includes an improvement advisor who is trained in the 
BSC methodology and provides the expertise to facilitate its implementation. 


Figure 2. Five Key Elements of a BSC 
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Source: CCL Project Team, based on Burroughs, J., Amaya-Jackson, L., Ebert, L., Agosti, J., Ake, G., & Goetz, K. (2013). The National Center for Child Traumatic Stress (NCCTS) learning 
collaborative model key elements checklist. Los Angeles, CA & Durham, NC: National Center for Child Traumatic Stress and Duke University Evidence Based Practice Implementation 
Center. 
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The change framework 


The BSC begins with the careful selection of a topic or issue and a specific shared goal for improvement. 
The Institute for Healthcare Improvement, the creator of the BSC, identifies three broad criteria for 
selecting a focal area for improvement (IHI, 2003). An appropriate focus for a BSC is a content area in 
which existing knowledge or evidence-informed practices are established but not widely used; results have 
been demonstrated in real world settings, so improvement is possible; and the current gap between what 

is known and what is done affects many people (such as children or families). A BSC is used to support the 
sustainable implementation of practices that are meaningful and will make a positive difference if improved. 


Topic selection then informs the development of what is called a change framework, which serves as the 
foundation for the collaborative. This framework for change is a document that specifies the focus of the 
improvement effort, as well as the evidence-based strategies or drivers of improvement for the desired 
outcome. The framework typically consists of four to ten core drivers of change that define the focus of 
the intervention and provide strategic guidance about what needs to be developed or shifted to achieve 
the desired BSC objectives. The change framework encompasses the research-informed EBPs that the 
intervention is designed to target, as well as the context and infrastructure necessary for successful 
implementation. One planning meeting is typically held to invite feedback on the change framework from 
key stakeholders (i.e., content and application experts and consumers). This meeting also provides an 
opportunity for the group of stakeholders to recommend complementary measures that teams may use to 
ensure progress and improvement. Additionally, the meeting may be used to identify possible solutions to 
the challenges that may arise during the BSC and help develop a cadre of advisors whose expertise can be 
tapped as the BSC proceeds. Planning and facilitating this meeting is a key implementation task. 


Most of the BSC Literature uses the term “change framework’ to refer to this guiding document. A core 
component of the change framework is the specification of the key drivers hypothesized as necessary 

to positively impact the desired outcome. Some descriptions of improvement collaboratives use the 

term “driver diagram” in addition to (or instead of) “change framework.” A driver diagram describes what 
elements must change in a system, how those elements must change to achieve the aims of the BSC, and 
what measures could help determine whether the changes result in improvements. For continuity when 
reviewing the literature, the term change framework is the language used throughout this paper. 


Multi-level inclusive teams 


The selection and enrollment of multi-level teams who will participate in the BSC begins with an application 
process used to select programs that demonstrate readiness for participating ina BSC and commitment 
to change. The application process may include activities designed to build readiness with potential 
candidate programs. Once the programs are selected and enrolled, multi-level teams are developed in 
each program, comprised of individuals from within the program—at all levels—who will participate in the 
BSC. The BSC method emphasizes the need for teams to include individuals closest to the work who will 
lead the implementation of new practices, parents or clients who are most impacted by the changes, and 
representation from senior leaders who support and are committed to change (Kilo, 1998). Senior leaders 
are expected to guide their teams in the collaborative and support the sustainability of the changes made 
through the BSC (IHI, 2003). The teams include individuals in the program or organization representing a 
variety of perspectives to promote buy-in to the process and the sustainability of changes made. 
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Expert faculty 


Once the change framework has been developed, implementation of the BSC includes hiring and training 
expert faculty who serve as subject matter/content experts. Individuals serving in this role are external to 
the teams and organizations participating in the collaborative. They receive training on the BSC and on 
their roles and responsibilities in the collaborative. The faculty help determine targets for improvement 

and provide ongoing support to teams in the selected content area. They plan and facilitate ongoing 
content training and coaching for the teams as needed. They work closely with project staff—including the 
improvement advisor who focuses on the process of making changes and helping teams conduct PDSA 
cycles to test and spread changes. The faculty and project staff together create a community of practice to 
provide ongoing support to teams participating in the collaborative. 


Shared learning environment 


The creation of a shared learning environment for the BSC includes the in-person learning sessions and a 
phase called “pre-work” that occurs before the learning sessions. Pre-work involves a series of activities to 
orient and prepare the participating teams to engage in the BSC process. The goal of pre-work is to clarify 
roles, responsibilities, expectations, and processes. Participants are given specific guidance and step-by- 
step tools to assist in team building, foundational content learning, data and measurement, and project 
management, and to facilitate program-wide buy in and communication. These activities are facilitated via a 
series of three to four preparatory calls or visits with program teams. Some collaboratives have emphasized 
the importance of using the pre-work period as an opportunity to collect baseline data for later comparison 
(Simanovski et al., 2014). 


Learning sessions typically consist of three or more face-to-face meetings during which the faculty and 
collaborative team participants exchange ideas. The purpose of the learning sessions is four-fold: 1) 
encourage and facilitate interaction between program teams, share learning and experiences with the 
PDSAs, and exchange ideas for sustaining changes and new practices; 2) plan within program teams 

and test strategies and formalize practice changes; 3) discuss specific themes and topics (practice, 
organizational, and methodological) identified as impacting successful implementation; and 4) teach and 
coach on content-specific practices and tools and on the BSC Model for Improvement. Planning for the 
learning sessions is an extensive and dynamic process to ensure that each event has a specific method for 
achieving its objectives, well-crafted materials, clear roles for faculty and other BSC implementation team 
members, alignment between topics and the teams’ needs, and useful and relevant tools and strategies 
that teams can test, refine, and spread. Although the topic of each learning session may vary, the BSC 
outlines a set of guidelines for each session. The first learning session focuses on discussing the vision of 
the BSC and the critical changes necessary for improvements in the identified shared aim. The second 
focuses on assessing and discussing progress across each team’s goals relative to the overall aim. The 
third and subsequent learning sessions are meant to solidify newly learned EBPs and/or improvements into 
sustainable change for individuals and programs/organizations. Participants share lessons learned and 
disseminate knowledge and strategies for sustainability during these learning sessions. In addition, a fourth 
learning session may be included to reinforce participants’ plans for sustainability. 


The shared learning environment is further supported during the action periods (the periods of time 
between learning sessions) through a variety of strategies, including phone conferences, an online 
workspace, monthly team reports, emails, visits or calls, and progress assessments (Ebert et al., 2012; IHI, 
2003). These supports help teams 1) continue and complete cycles of new and previously tested PDSAs, 2) 
formalize and implement new practices, and 3) spread successes more broadly across the program and the 
entire learning collaborative. Overall, the action periods are intended to support the spread of successful 
changes and promote a broad change in organizational culture. 
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The faculty and project staff typically host topic-focused calls each month that include participants from 
all teams. These one-hour calls are often co-facilitated by a participating team, giving them the opportunity 
to share their successes and lessons learned while other participants can engage in conversation about 
specific changes. In addition to the monthly all-team calls, there are also monthly “affinity team” calls that 
bring together similar roles from the various teams (e.g., senior leaders, teachers, non-teaching staff, parent 
leaders, etc.). These monthly calls allow the affinity teams to deepen their relationships across teams in the 
context of sharing role-specific experiences. A final venue for supporting the shared learning environment 
is a protected online project site. This site has a document repository to allow faculty and teams to share 
resources, tools, and other documents; a message board that allows for threaded conversations among 
and across team members; and a section dedicated for teams to post their PDSAs (see below for more on 
PDSAs and the model for improvement). 


Model for improvement 


During action periods, teams engage in rapid tests of change in their practices. Action periods typically 

last 3 to 5 months, during which teams collect data to assess the effects of these changes. This process 

is guided by the model for improvement, based on Deming’s (1986) process improvement work, used to 
help organizations reflect on their goals for change, their metrics for knowing whether changes result in 
improvements, and their strategies for identifying changes to test. Key changes are then tested using the 
PDSA cycle, in which teams plan to test a specific change, conduct the work necessary for changing the 
practice, study the effects of that change on their work using quantitative measures, and act by abandoning 
unsuccessful changes—all while making adjustments to refine changes or growing changes to test in 
different contexts (IHI, 2003). This cycle of testing, adapting, and implementing changes over time is 
expected to promote changes that result in improvement. 


The model for improvement is used as the touchstone to guide all improvements tested by teams. The 
three questions asked together repeatedly are: 1) What are we trying to accomplish? 2) How will we know 
that a change is an improvement? and 3) What changes can we test that will result in improvement? The 
first question is typically answered in reference to the change framework, based on the change themes 
identified. For the second question teams will look to the collaborative metrics. For the final question, 
teams will again look to the change framework as they get ideas from the change concepts and strategies 
identified at the expert meeting. 


The five elements together 


Overall, the five elements of the BSC methodology support and sustain the implementation of evidence- 
based practices in the real-world contexts in which people work. Participation in the BSC requires that 
multi-level teams be engaged and committed to the improvement process. For example, teams must be 
prepared to attend learning sessions and use the model of improvement to test and implement changes. 
Participating programs may need to make adjustments to staffing and workload to accommodate team 
members’ responsibilities in the BSC. Participation in the BSC is, in fact, an intensive endeavor for programs. 
When considering the potential of using the BSC in the ECE field, it is important to address questions about 
the feasibility of the model in ECE programs and the extent to which any elements of the BSC would need 
to be adapted for the ECE context. As noted in the introduction to this paper, the CCL Project is being 
conducted to address these feasibility questions. The next section provides an overview of the literature 
examining the process, outputs, and outcomes of the BSC. 
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IV. A Review of the Literature: BSC Outputs and Outcomes 


Two decades of research have documented the effects of implementing the BSC methodology (e.g., IHI, 
2003; Nadeem et al., 2013). Although the majority of BSC literature derives from the health care field, a few 
studies have also assessed the effects of implementing the BSC methodology in the fields of child welfare 
and mental health (e.g., Agosti et al., 2013; Conradi et al., 2011; Ebert et al., 2012). In most articles reviewed in 
this section, participating programs include hospitals, health centers, nursing homes, mental health centers, 
and child welfare agencies. 


The reviewed studies examined BSC implementation across multiple teams (ranging from four to 141 team), 
often located within multiple programs. Of the 49 BSC studies reviewed, six reported on BSC outputs 

and 34 reported on outcomes, which are summarized in Table 1. Outputs of the BSC primarily include 
participation in the BSC learning sessions and monthly calls, and details about implementation activities 
such as the number of staff trained on a specific practice or number of site visits conducted by the BSC 
faculty and/or project staff. Outcomes of the BSC are discussed in the literature in terms of immediate or 
“proximal” outcomes and longer-term “distal” outcomes. Proximal outcomes were discussed in the literature 
in different ways. Short-term outcomes were noted at the levels of (1) the organization (e.g., infrastructure to 
support change in practice); (2) the community, or outcomes across participating teams; (3) the practitioner 
(e.g., changes to practice, knowledge, and skills); and (4) the potential recipient of the intervention, such as 
a child or patient. Although less prevalent in the literature, long-term outcomes were noted in a few studies, 
and generally included outcomes related to the spread or sustainability of new practices. 


This section reports on the outputs and the proximal and distal outcomes of the BSC reported in the 
literature. Most studies reviewed focused on assessing proximal outcomes at the participant level (76%) 

and recipient level (71%); fewer studies assessed proximal outcomes at the organization (21%) or team level 
(12%). Although the number of studies reporting proximal outcomes varies by outcome level, the majority 
(97%) reported at least one proximal outcome, while about one-third (36%) reported at least one distal 
outcome. Furthermore, the outcomes in the BSC literature were largely assessed using self-report strategies 
(79%; e.g., self-assessment rubrics related to the EBP, surveys, interviews, focus groups) rather than a direct 
measure of changes in practice (55%; e.g., a review of medical records to assess changes in practices, such 
as referrals, examinations, assessments) or direct measures of recipient outcomes (e.g., disease status, 
remission, infection rates). Of the 34 studies that reported outcomes, only three included a control group. 


It is important to note discussion in the literature related to challenges in measuring outcomes and changes 
that may result from the BSC or other quality improvement methodologies (Mittman & Salem-Schatz, 2012). 
Several factors contribute to this challenge. First, improvement approaches often have multiple complex 
goals. Not all studies reported on or measured each proximal and distal outcome. Therefore, it is difficult 

to ascertain from the literature whether the outcome areas that were less frequently reported were: (1) not 
measured; (2) if there were challenges in how the outcomes were measured; or (3) if outcomes were simply 
not reported because no change was observed. Second, participation outputs are seen as precursors to 
changes in practice. If participation throughout the duration of the BSC was not as expected, the expected 
outcomes may also not have been as expected. However, participation rates were not always measured or 
reported in the studies reviewed. Among the studies reviewed, negative or null results were not common; 
however, one study that did not result in positive outcomes provided an insight into the complexity around 
measuring BSC outcomes: “[w]Je do not know whether the Collaborative failed [to meet the identified 
outcome] because participants were not engaged actively enough, because of the types of changes that 
were implemented, or because of the way that they were implemented” (Landon et al., 2004, p. 895). These 
complexities are important to consider when reviewing the BSC output and outcomes identified in the 
literature. 
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In this section, findings reported in the BSC literature are summarized according to participant outputs and 
both proximal and distal outcomes at the organizational, team, practitioner, and recipient levels. 


Participation in the BSC: BSC Outputs 


Successful implementation of the BSC methodology requires extensive participation of the selected 
team—typically comprised of individuals in a range of roles, including front-line staff, supervisors, and 
administrators. Several components of the BSC require participant engagement, including learning 

sessions and other BSC communication mediums (e.g., conference calls, email, intranet, progress reports) 
expected to facilitate progress during action periods. Participation in the BSC is important to consider when 
examining the outcomes of the BSC, as the extent to which the BSC is being implemented as intended likely 
has implications for the outcomes achieved. Of the 49 studies reviewed, only six included any measure of 
participation. In the studies reviewed, participation across the BSC components varied, particularly across 
the elements expected to support progress during the BSC action periods. 


Learning sessions. Participation rates in BSC learning sessions were fairly high across the studies that 
reported on BSC participation. In these studies, the majority of the team members attended all or most 
learning sessions (Arbour, Yoshikawa, Atwood, Duran Mellado, Godoy Ossa, Trevino Villareal, & Snow, 2016; 
Ebert et al., 2012; Haine-Schlagel et al., 2013; Homer et al., 2005;). One study also found that a majority of 
participants were satisfied with the learning sessions (95%), perceived them as worthwhile (96%), and felt 
that having three two-day learning sessions was more useful for their implementation efforts compared to 
a single training of a similar length (93%; Ebert et al., 2012). In this study, participants saw value in attending 
the BSC activities even if they did not attend every learning session. 


Action periods. BSC teams use PDSAs during action periods to facilitate implementation of an EBP. One 
study found that, across an 18-month initiative, teams completed between 3 and 14 PDSAs (median = 6), 
many of which tested a series of related changes over time (Ebert et al., 2012). However, the majority of 
participants reported that they did not share all of their PDSAs on the learning collaborative online share 
site (which was reviewed by the research team to track participation); therefore, the number of PDSAs 
conducted by the teams may have been underestimated. Over half of participants (53%) in the study by 
Ebert and colleagues (2012) reported that the PDSAs were very/extremely useful, while another 30 percent 
reported that PDSAs were at least somewhat useful to their implementation efforts. In a second study, the 
research team estimated that each team completed between 15 and 20 cycles, although no formal count 
of PDSAs was conducted across the 13-month collaborative (Glasgow et al., 2002). A third study reported 
that, across a 21-month initiative, coaches assessed the implementation of PDSAs by assigning a score 
based on the extent to which teams had conducted tests of change, attained measurable progress, and 
achieved goals. By the end of the collaborative, over 93 percent (n = 1,574) of the PDSAs attempted by the 
141 participating teams were successfully implemented, demonstrating that they had achieved the aims 
set forth by the PDSA. By contrast, 2 percent (n = 29) of PDSAs were abandoned and 6 percent (n = 94) 
were described as a “work-in-progress,” characterized by multiple tests of change with some measurable 
progress (Bidassie, Davies, Stark, & Boushon, 2014). 


During action periods, BSC teams from each program are supported through conference calls with other 
program teams and expert faculty members. One study that examined participation in conference calls 
found that participation was mixed, with 80 percent of supervisors and 40 percent of clinicians participating 
in at least four of the seven collaborative calls (Ebert et al., 2012). Ebert and colleagues (2012) noted 

that some of the variation in participation could be explained by the fact that few participants reported 

that conference calls involving all participants were very useful (19%). However, at least half (i.e., 50% 

of senior leaders and 68% of supervisors) found that affinity group conference calls—those limited to 
participants within the same role (e.g., senior leaders, supervisors)—were very or extremely useful. Another 
study noted that clinicians were often dissatisfied with conference calls, as the topics discussed were not 
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always relevant to their work (Lang, Franks, Epstein, Stover, & Oliver, 2015). These findings suggest that 
participants may be more likely to engage in calls they perceive to be directly relevant to their work and 
indicate that, when applicable, calls tailored to affinity groups may be most useful. 


Program teams are required to submit monthly reports during the action periods that summarize their 
progress on improvement metrics identified in the change framework. In one study that reported on 
progress report data, 42 percent of participating teams submitted progress data and 39 percent completed 
progress reports in any given month (Haine-Schlagel et al., 2013). Similarly, Ebert and colleagues (2012) 
indicated that, although 9 of 11 participating teams developed improvement metrics, about half of the teams 
(5) used these metrics consistently. While over 75 percent of administrators found the metrics to be useful 
to their implementation efforts, only 40 percent of clinicians and less than 15 percent of supervisors agreed 
they were useful (Ebert et al., 2012). 


Summary. Reports of variation in participation across the BSC components highlight the need for greater 
understanding of the situational and participant characteristics necessary for ensuring high participant 
engagement and, in turn, how participation may affect the intended outcomes of the BSC. Although 
variation in participation is evident, participants generally were satisfied with the BSC components. Results 
also suggest that perceptions sometimes varied by team members’ roles in the organization, highlighting a 
gap in our understanding of how different groups of participants experience the BSC. Future studies should 
not only track and report on participation in BSC components by role, but also examine the extent to which 
participant engagement affects the intended outcomes of the collaborative. 


BSC Outcomes 


Of the 49 studies reviewed, 34 studies reported on outcomes of the BSC. Table 1 summarizes findings from 
the 34 studies related to both proximal and distal outcomes. The following section provides a summary and 
analysis of the findings at the organizational, team, practitioner, and recipient levels. 


Proximal outcomes 


Organizational outcomes. Few studies (7 of 34, or 21%) assessed organizational outcomes following 

the implementation of a BSC. The seven studies reporting on organizational outcomes employed both 
quantitative and qualitative data collection strategies, finding that the BSC was a useful mechanism for 
developing the infrastructure to support ongoing quality improvement and developing a culture that 
promotes the use of selected evidence-based practices in the healthcare sector (Backer, Kiser, Gillham, & 
Smith, 2015; Bitton et al., 2014; Simanovski et al., 2014) and in the field of trauma-informed care (Ebert et 
al., 2012). For example, in one study, participants reported that the BSC approach supported an increased 
commitment to high-quality implementation and the diffusion of EBP implementation to other communities 
(Ebert et al., 2012). In another study, participants reported that the collaborative helped them develop the 
processes and infrastructure to support ongoing quality improvement and promote a culture of safety within 
their organization (Simanovski et al., 2014). 


Team outcomes. Four studies (12%) reported team-level proximal outcomes of the BSC. These studies also 
assessed team-level outcomes using self-report (e.g., surveys/interviews) strategies. Overall, the studies 
find some evidence that the BSC can facilitate the strengthening of team-level processes. Two studies 
reported positive team outcomes, documenting that teams worked more effectively after participating in 
the BSC; teams also perceived that the BSC supported improved team communication and facilitated the 
sharing of ideas, lessons learned, and knowledge with peers across programs and organizations (Bitton et 
al., 2014; Chin et al., 2004). The other two studies indicated that team-level outcomes were mixed, reporting 
some positive team-level effects while also noting challenges in meeting more difficult team-level outcomes 
(e.g., improving patient handovers; Bidassie, Davies, Stark, and Boushon, 2014). 
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Practitioner outcomes. The majority of studies that reported on outcomes (25 of 34, or 76%) reported 
practitioner-level outcomes of the BSC. These outcomes were assessed by asking practitioners to report on 
their practices before and after the BSC (48%), directly assessing practices using medical records (44%), 
or both (8%). Of the studies that reported practitioner outcomes, most (20 of 25 studies) reported positive 
outcomes. For example, the BSC was related to improved staff attitudes toward trauma-focused cognitive 
behavioral therapy—an evidence-based practice (Lang et al., 2015). In one study conducted in Chile, ECE 
programs used the BSC to support teachers’ instructional practices (Arbour et al., 2016). At the end of 

the study, teachers reported that their roles, and the level of respect they received from peers and senior 
leadership, improved as a result of their participation in the BSC and the skills they developed (Arbour et 

al, 2016). Other studies described how the BSC method was used to successfully increase adherence to 
evidence-based practices related to a number of focus areas, including adolescent permanency, diabetes 
care, pediatric preventative care, trauma-focused therapy for children, and chemotherapy delivery (Agosti 
& Morrill, 2007; Chin et al., 2004; Daniel et al., 2004a; Duncan et al., 2015; Ebert et al., 2012; Simanovski 

et al., 2014). Four studies had mixed results and found that, although the BSC resulted in positive changes 
in practitioners’ skills and approaches, most expected outcomes (e.g., reduction of infection in preterm 
infants) were not observed or did not demonstrate significant improvement (Baier et al., 2004; Glasgow et 
al., 2002; Kaplan, Lannon, Walsh, & Donovan, 2011; Mercier et al., 2007). These four studies employed direct 
assessments of medical charts to document changes in practices related to infections in preterm infants, 
diabetes and heart failure, pain management, and newborn preventative services. 


Overall, the BSC literature provides evidence that the methodology can facilitate the implementation of 
evidence-based practices. However, the mixed findings reported in studies using direct assessments of 
practice suggest the need for more objective evidence of changes in practice linked to BSC participation. 


Recipient outcomes. Proximal outcomes at the recipient level have been assessed almost as frequently 

as outcomes at the practitioner level (Table 1). Of the 34 studies that reported on outcomes, 24 (71%) 
examined outcomes at this level, focusing on patients or participants. Most of these studies employed 

a direct assessment of recipient outcomes (63%); however, some used self-report measures (25%; e€.g., 
surveys or interviews) or a combination of both (13%). Of these 24 studies, 19 (79%) reported positive 
outcomes for recipients. For example, the study that examined the use of the BSC to improve instructional 
practices in early care and education settings in Chile found positive impacts on child outcomes. Analyses 
using propensity score matching demonstrated that children in classrooms whose teachers participated in 
the BSC and received professional development on instructional strategies had larger increases in language 
skills (vocabulary, effect size 0.31, p<0.05) than children whose teachers received professional development 
on instructional strategies alone (Arbour et al., 2016). 


Three studies (13%) reported mixed findings at the recipient level (Asch et al., 2005; Daniel et al., 2004b; 
Flamm, Berwick, & Kabcenell,1998; Neily, Howard, Quigley, & Mills, 2005) and two (8%) found null effects 
(Homer et al., 2005; Landon et al., 2004). For instance, studies found reductions in rates of patients with pain 
(Baier et al., 2004), infections in infants related to bacterial pathogens (Kaplan et al. 2011), post-traumatic 
stress disorder and depression symptoms (Lang et al., 2015), increases in patient self-monitoring (Glasgow et 
al., 2002), and family attendance at community-based services (Haine-Schlagel et al., 2013). However, other 
studies found no differences in the proportion of patients with a suppressed viral load (Landon et al., 2004) 
and asthma attacks (Homer et al., 2005) when comparing individuals in the intervention group to those in 

a control group. As noted with regard to practitioner outcomes, Landon and colleagues (2004) found that 
health care practices did not improve after implementing the BSC and recipient-level outcomes also were not 
affected. However, the study examining asthma attack care (Homer et al., 2005) only measured recipient- 
level outcomes, making it difficult to determine whether the BSC led to improvements at the organization 

or provider level that did not translate into positive recipient-level outcomes. Taken together, however, the 
majority of the studies indicate that BSC implementation can ultimately promote benefits at the recipient level. 
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Distal outcomes 


Of the studies reviewed, 12 (35%) examined longer-term effects of the BSC on organizational-, practitioner-, 
or recipient-level outcomes. Seven (21%) reported on distal organizational outcomes, seven (21%) reported 
on distal practitioner outcomes, and three (9%) reported on distal recipient-level outcomes. These studies 
also largely employed self-report measures (e.g., surveys and interviews). Although fewer studies focused 
on distal outcomes, the pattern of findings is consistently positive. All seven studies reporting on distal 
organizational outcomes found positive results (Bitton et al, 2014; Boushon et al., 2006; Conradi et al., 

2011; Daniel et al., 2004b; Ebert et al., 2012; Miller et al., 2008; Neily et al., 2005), as did the seven studies 
reporting on distal practitioner outcomes (Backer et al., 2015; Boushon et al., 2006; Daniel et al., 2004b; 
Ebert et al., 2012; Haine-Schlagel, 2013; Lang et al., 2015; Simanovski et al., 2014). For example, one study 
reported that the organization spread the use of the collaborative methodology to additional community 
clinics, expanded the use of the BSC to other focus areas, and required all physicians to participate in the 
disease management program facilitated by the BSC (Daniel at al., 2004b). Two of the studies examining 
distal recipient outcomes (Backer et al., 2015; Neily et al., 2005) reported positive findings, while the 

other reported mixed findings (Daniel et al., 2004b). While the two studies that reported positive findings 
indicated sustained gains over time after the BSC ended, the third study indicated that, although the gains 
made during the BSC process had been maintained, they were not at the same level as during active 
participation in the collaborative. 


Summary 


A review of the BSC literature indicates consistently positive findings for organizational, team/practitioner, 
and recipient outcomes in both the short term and long term. In general, however, there is less evidence for 
long-term effects of the BSC across the health, child welfare, and trauma-informed care literature. 


A previous review of the BSC literature specifically focused on outcomes for patients with chronic heart 
disease (Newton, Davidson, Halcomb, Denniss, & Westgarth, 2006) and found that, across 23 studies,® “the 
Collaborative methodology, when implemented and conducted according to key conceptual principles, 

has significant potential to improve the outcomes of patients” (p. 147). The BSCs that received mixed 

or null results for recipients (both in the short term and in the longer term) have targeted very complex 

and intractable conditions such as chronic heart failure and HIV (Asch et al., 2005; Landon et al., 2004). 
While changes in these conditions may be particularly difficult to achieve, patients may still benefit from 
the positive and sustained changes in organizational climate, infrastructure, and more consistent use of 
evidence-based practices by the practitioners assigned to their care. 


Overall, the literature examining the implementation and outcomes of the BSC indicate that the methodol- 
ogy has consistent evidence of improving practices in health care, child welfare, and mental health service 
provision by facilitating the implementation of EBPs and building the infrastructure to support quality im- 
provement efforts (e.g., Backer et al., 2015; Duncan et al. 2015; Ebert et al., 2011; Glasgow et al., 2002; Haine- 
Schlagel et al., 2013; Russ, Hanna, DesGeorges, & Forsman, 2010). Although outcomes at the recipient level 
were either not reported or mixed, the overall findings from this review indicate that the BSC model holds 
promise for improving the quality of organizational processes and individual practice in the early care and 
education sector. 


5 Only three articles in our current review overlap with this previous review (i.e., Daniel et al., 2004; Flamm et al., 1998; Glasgow et al., 2002). This 
may be due in part to our focus on primarily searching for literature available through social science databases (e.g., EBSCO, PsycINFO). The 
Newton et al., article is published in a medical journal and only appeared in a secondary search conducted in Google Scholar that looked beyond 
the search terms outlined in our original scan of the literature. 
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Table 1. Summary of Literature Assessing the Outcomes of the BSC Model 


Organizational Team Practitioner Recipient Organizational Practitioner Recipient 
Agosti & Morrill, 
9 Adolescent permanency + + + 
2007 
Arbour et al., 2016 ECE programs in Chile + + 
Asch et al., 2005 Chronic heart failure care +/- 
Resilience-informed mental 
Backer et al., 2015 . + + + + + 
health services 
: Pain-management processes 
Baier et al., 2004 7 anagemcn pace +/- + 
of care 
Bidassie et al., 2014 | Patient-centered medical home +/- + 
Primar re practi n 
Bitton et al., 2014 a oe epreunceand + + + + 
education 
Boushon et al., 2006 | Access in primary care + + + + 
Chin et al., 2004 Diabetes care + *t + 
Depression identification and 
Chung et al., 2011 Spies Siar idenMncatons + 
treatment 
. Trauma-informed welfare 
Conradi et al., 2011 ‘ + 
practice 
Pediatric inflammator wel 
Candabera. ow: | <a nlammaty howe + ‘ 
disease care 
Daniel et al 2004a | Diabetes care + + 
Daniel et al 2004b | Diabetes care + + + +/- 
Duncan et al., 2015 Pediatric preventative care + 
Evidence-based trauma 
Ebert et al., 2012 treatment for children in + + + + 
community practice settings 
Flamm et al., 1998 Reducing cesarean birth rates +/- 
Glasgow et al., 2002 | Diabetes and heart failure care +/- + 
Homer et al., 2005 Childhood asthma treatment - 
Acute ischemic strok 
Hseih et al., 2016 Gute leenemie 2H ane + 
management 


Key: + indicates positive findings (at least 75% of measured outcomes demonstrated improvement); +/- indicates mixed findings (less than 75% of measured outcomes demonstrated improvement); - indicates null findings. Note: Of the 49 studies 
identified for inclusion in this literature review, 34 reported on outcomes of the BSC. Only these 34 studies are represented in this summary table. 
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Table 1 cont. Summary of Literature Assessing the Outcomes of the BSC Model 


Organizational Team Practitioner Recipient Organizational Practitioner Recipient 
Reduction of late-onset sepsis 
Kaplan et al., 2011 : . +/- + 
in preterm infants 
Katzelnick et al., : 
Depression treatment + + + 
2005 
Landon et al. 2004 | HIV treatment + = 
ing ctal odie Trauma-focused cognitive ¢ ” re 
9 7 behavioral therapy 
Mangione-Smith oats 
analone-Siititer Pediatric asthma care + + + 
al., 2005 
. Preventative services for 
Mercier et al., 2007 +/- 
newborns 
Racial disproportionality and 
Miller & Ward, 2008 | disparate outcome in child +/- + + 
welfare 
F Reducing falls and fall-related 
Neily et al., 2005 Bivete + + 
injuries 
Shi FeMiiata) Reduce inappropriately- 
Writi i ; f + + 
deg Pomimnitiee scheduled births 
2010 
Russ et al., 2010 Newborn hearing screening + + 
Shafer et al., 2008 Organ donation + + 
i ki I, : 
Sinanovesvetd Chemotherapy delivery + + + 
2015 
Y | I, eid 
esecaaca HIV transmission program + +/- 
2010 
Total + 7 2 20 19 i 6 
Total - fe) fe) 1 2 fo) fe) fe) 
Total +/- fe) 2 4 3 (0) (e) 1 
Overall Totals 7 4 25 24 7 6 4 


Key: + indicates positive findings (at least 75% of measured outcomes demonstrated improvement); +/- indicates mixed findings (less than 75% of measured outcomes demonstrated improvement); - indicates null findings. Note: Of the 49 studies 
identified for inclusion in this literature review, 34 reported on outcomes of the BSC. Only these 34 studies are represented in this summary table. 
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V. Considerations and Next Steps for the CCL Project 


The review of the BSC literature indicates consistently positive findings for organizational and team/ 
practitioner outcomes in both the short term and long term. The literature also indicates that the BSC has 
consistent evidence of improving outcomes and improving the implementation of quality practices in health 
care, child welfare, and mental health service provision and building the infrastructure to support future 
quality improvement efforts (e.g.., Backer et al., 2015; Duncan et al. 2015; Ebert et al., 2012; Glasgow et al., 
2002; Haine-Schlagel et al., 2013; Russ et al., 2010). Based on these findings, the BSC methodology presents 
a unique approach to quality improvement worthy of feasibility testing in early care and education settings. 
However, there are more mixed or variable findings for the ultimate recipients of the BSC’s focused efforts. 
Additionally, there is less evidence for long-term effects of the BSC across the health, child welfare, and 
trauma-informed care literature. Taking these findings into account, there are several considerations for 
next steps for the CCL Project. 


Assess organizational readiness. During program selection, a process should be in place to screen 
potential program participants along organizational dimensions such as culture, infrastructure, capacity, 
motivation, sufficient time to engage in the BSC, and engagement of key leaders. While it may not be 
possible to predict or account for all features that may lead to a good fit between the BSC and selected 
programs (i.e., financial stability over time), assessing these features of organizational readiness at the 
start—and perhaps periodically throughout BSC implementation—may improve our understanding of the 
feasibility of using a BSC in the ECE programs. 


The research reviewed in this section also indicates that practitioners who have used the BSC were 
motivated to do so because they recognized significant inadequacies in their current ways of implementing 
quality improvement practices or EBPs in their care settings. Some were also motivated to use a new 
approach because of the BSC’s intent to connect senior leadership with the practitioners charged with 
implementing new ways of working. In the CCL Project, it may be important to examine the extent to 
which programs considering the BSC also perceive significant deficiencies in their current approaches to 
quality improvement, and whether these perceptions are significant enough to prompt them to seek out 
amore intensive approach that they can commit to and sustain over time. These insights also suggest the 
need to understand these perceptions from both a bottom-up and top-down perspective, to ensure that 
a systematic and comprehensive approach is warranted and would be welcomed from those tasked with 
implementing new practices and from senior leaders. 


Maximize the potential of the pre-work period to set participants up for success. Readiness to engage 
in the BSC and the ability to use the pre-work period to maximize the preparation of the teams was noted 

in various ways in the literature. Organizational readiness is therefore not just a selection criterion for 
participation in a BSC, but also an expected outcome of the BSC. In the CCL Project, it will be important to 
use the pre-work period to build teams’ capacity to ensure that selected programs have the time, resources, 
knowledge, and motivation to engage in the BSC. The pre-work period was also noted as an important 
opportunity in the BSC process to help build team capacity for data collection and self-assessment. 


Examine facilitators of change. Researchers have argued that there is limited information on the attributes 
of the BSC most likely to facilitate change, therefore making it difficult to maximize the effectiveness and 
transferability of this model as a strategy for driving change (Nadeem et al., 2013). As noted in the literature, 
components of the BSC model may affect both proximal and distal outcomes for children, practitioners, 

and organizations. In the design of the feasibility study, we will include a focus on the core elements of the 
BSC model: (1) the change framework; (2) multi-level inclusive teams; (3) expert faculty and coaches; (4) 

a shared learning environment; and (5) the model for improvement. We will also focus on the outcomes 
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associated with positive change in individual practice and organizational structures and culture, but will not 
address child and family outcomes. These will be addressed, however, in the evaluation study design that will 
be completed for the project. 


Measure BSC outputs such as participation. The review of the literature on outputs of the BSC 

(i.e., participants’ engagement in all aspects of the BSC process) concluded that there is a gap in our 
understanding of how different groups experience the BSC and how their engagement in the BSC 

process could be maximized. As part of the feasibility study for the CCL Project, we will focus especially 

on the experiences of Head Start and child care programs to determine whether different supports are 
needed for successful engagement in the BSC process. Thus, a focus on BSC outputs will be critical in 

the data collection plans for the feasibility study. However, we will also plan to examine several proximal 
outcomes of a BSC for organizations and individuals—specifically positive change in individual practice and 
organizational culture within early care and education settings. 


Use objective measurement approaches. Researchers argue the need for more objective evaluations 
of the BSC model that can assess its ultimate impact, not only at the individual level but also in relation to 
cost effectiveness (Mittman, 2004; Nadeem et al., 2013). This critique will be important to consider when 
planning for a cost study as part of the evaluation study design. In addition, we intend to use multiple 
methods (e.g., direct assessments, participant or recipient reports, observations, review of administrative 
data) to gather information on outcomes of interest in an attempt to triangulate the evidence for practice 
change. 


Discuss expectations related to long-term outcomes. We note the general lack of assessment of distal 
outcomes of the BSC model. This will be critical to consider when designing the outcomes study for this 
project. Specifically, as noted above, it will be important to look for evidence in sustained organizational and 
individual practice change by examining organizational or individual change both early (i.e., at baseline, just 
as the BSC process is beginning) and later (i.e., at the end of a 12-month BSC process and also, if possible, 
12 months later). It would also be worthwhile to attempt to observe the functioning of learning groups, or use 
of PDSAs, after the end of the BSC process. 


Next Steps for the CCL Project 


The purpose of this literature review was to examine the evidence base of the BSC to better understand 
its outputs and outcomes, as well as the contextual factors that may support success. Taken together, 

the literature suggests that the activities in the CCL Project have the potential to address a pressing need 
in the ECE field to enhance and even transform current approaches to quality improvement. The project 

is a first step in testing the feasibility of an innovative method to inform the design of a future evaluation 
study. The lessons learned from the literature suggest that the CCL Project may need to support buy- 

in and engagement in the BSC at multiple levels of the system; ensure supports for implementation, 
including training, supervision, and data systems; develop strategies for building capacity to engage in the 
improvement process at all levels; examine ongoing incentives for supporting participation in the initiative; 
and identify strategies for trouble-shooting common problems that arise. These considerations will be 
incorporated into the next phase of the CCL Project, which includes a feasibility study to examine the 
implementation of the BSC with select ECE programs. The results of this study will help determine whether 
the BSC holds promise as a structured approach to improvement in ECE settings and the feasibility of 
implementing the BSC in different types of early care and education programs. Results will also be used to 
inform planning for future research related to quality improvement in ECE programs. 
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